The present invention relates generally to the field of transmitting data over networks and particularly relates to streaming media technology.
Transmitting audio, video and combined data over networks, including the Internet, has become a routine aspect of modem life. In a typical scenario, a user downloads an application program (such as xe2x80x9cplayerxe2x80x9d software) to a hard drive on a personal computer, then downloads an entire data file (such as an audio or video file), then plays the file. Although user only needs to download the player once, the user must fully download each audio or video file before it can be played.
Due to the advent of xe2x80x9cstreaming mediaxe2x80x9d or xe2x80x9cmultimedia streamingxe2x80x9d technology, users no longer need to download an entire data file before playing it. Instead, an on-line content provider transmits discrete packets of data, which are normally compressed, in a continuous sequence or xe2x80x9cstream.xe2x80x9d If a user has already downloaded or otherwise installed an application program, the user may play each packet as soon as the packet is received and decompressed.
Although prior art streaming media technology represents a significant advancement over the prior art, a user will often experience significant delays if all data are not in the same type of industry-standard format or if the entire application or player program is large or not installed in the user""s machine. If these delays are significant, a user will not be satisfied with the resulting slow response. The user may decide not to use the associated software. With regard to use at an Internet web site, if delays are significant the user may simply browse another web site, which could mean lost revenue for the web site owner.
It would be of great benefit if users could download and play streams of packets which include multiple data types and those which require large application programs to play back, while keeping the initial latency before displaying the multimedia data to the screen to a minimum.
It would be of significant value if raw timed multimedia data could be quickly encoded into such a scheme because such multimedia data are a natural output format of multimedia design tools. Additionally, it would be of great value if the streamed packets could be dynamically encoded for different bandwidths and transmission times.
The present invention includes various methods and devices for implementing compression, streaming, and decompression of multimedia data packets between a client and a server program. These methods can encode directly and very efficiently from raw timed multimedia data. The playback offers both very low initial latency and unlimited playback length because of the unique way data and instructions are merged together.
In some embodiments, a time code or xe2x80x9ctime stamp,xe2x80x9d multimedia data and instructions are included in a single data packet which is part of a stream of data packets. The time stamp may indicate a time at which a data packet shall be transmitted or a time at which corresponding instructions shall be executed.
In this disclosure, an instruction may be termed a procedure call, a remote procedure call (an xe2x80x9cRPCxe2x80x9d), a function call, or may have other names. Such instructions typically include one or more commands relating to the reproduction of multimedia data. In many embodiments of the present invention, the instructions are used by xe2x80x9cplayerxe2x80x9d software installed on a client computer to perform operations on the multimedia data.
In some embodiments, the data packets also include parameters of the associated data. In still other embodiments, a data packet will include instructions and a reference to data which are not included within the same data packet. Such data may reside on a computer which receives the data packet.
In some embodiments, some data packets will not have time stamps, but will be associated with other data packets which have time stamps. In some such embodiments, the instructions in a data packet with no time stamp will be executed at approximately the same time at which instructions in an associated data packet which has a time stamp will be executed. In other embodiments, the instructions in a data packet with no time stamp will be executed before or after the time at which instructions in an associated data packet which has a time stamp will be executed.
According to the present invention, multimedia data are organized into compressed data packets. A set of encoding routines transforms a set of instructions and multimedia data into a set of time-stamped procedure calls which are transferred with corresponding portions of the multimedia data upon which the instructions will operate. In some embodiments, data parameters are also included in the data packets. Each procedure call is later converted to an RPC: the parameters are organized into data packets and all data needed for the RPC are time-stamped and stored. All RPC""s are stored in sequence, sorted by time stamp. In some embodiments, the encoding process compresses the stored data. In some embodiments, the resulting data packets are compressed again, e.g., by using a lossless dictionary compression method.
The process of organizing data into packets may be referred to in this application as xe2x80x9cmarshaling.xe2x80x9d This term is not limited to its common meaning in the art, but includes any convenient way of organizing data into data packets according to the present invention.
A method according to one embodiment of the present invention includes the steps of: (1) determining a transmission time, based transmission data which include information regarding a speed of a transmission medium, for a plurality of multimedia data which correspond to a plurality of instructions; (2) assigning time stamps to at least a portion of the instructions, the time stamps corresponding to the determined transmission times; and (3) arranging the time stamps, the instructions and the multimedia data into a sequence of data packets for controlling a player to reproduce the multimedia data.
A device according to an embodiment of the present invention includes: (1) an apparatus for determining a transmission time, based upon transmission data which include information regarding a speed of a transmission medium, for a plurality of multimedia data which correspond to a plurality of instructions; (2) an apparatus for assigning time stamps to at least a portion of the instructions, the time stamps corresponding to the determined transmission times; and (3) an apparatus for arranging the time stamps, the instructions and the multimedia data into a sequence of data packets.
Another method according to the present invention includes the steps of: (1) receiving data packets; (2) de-marshalling the data packets into time stamps, instructions and multimedia data; (3) determining an execution time, based upon the time stamps, for each of the instructions; and (4) executing the instructions, thereby reproducing the multimedia data.
Yet another method according to the present invention includes the following steps: (1) receiving a data packet; (2) de-marshalling the data packet into a time stamp, an instruction and a data code which corresponds to data stored in memory accessible to the local computer; (3) determining an execution time, based upon the time stamps, for the instruction; (4) retrieving the data which corresponds to the data code; and (5) executing the instruction, thereby reproducing the data which corresponds to the data code.
Some methods according to the present invention involve the asynchronous storage and retrieval of time-stamped data packets. In some such embodiments, time stamps, instructions and multimedia data are stored in a first device at a first time and are requested by a second device at a second time.
An apparatus according to another embodiment of the present invention includes: (1) an apparatus for receiving data packets; (2) an apparatus for de-marshalling the data packets into time stamps, instructions and multimedia data; (3) an apparatus for determining an execution time, based upon the time stamps, for each of the instructions; and (4) an apparatus for executing the instructions, thereby reproducing the multimedia data.
An apparatus according to still another embodiment of the present invention includes: (1) an apparatus for receiving a data packet; (2) an apparatus for de-marshalling the data packet into a time stamp, an instruction and a data code which corresponds to data stored in memory accessible to the local computer; (3) an apparatus for determining an execution time, based upon the time stamps, for the instruction; (4) an apparatus for retrieving the data which corresponds to the data code; and (5) an apparatus for executing the instruction, thereby reproducing the data which corresponds to the data code.
A data packet for controlling a computer to reproduce multimedia data according to some embodiments of the present invention includes: (1) a data code which corresponds to data stored in memory accessible to a computer which receives the data packet; (2) an instruction to player software stored on the computer regarding reproduction of the data; and (3) a time code indicating when the instruction will be performed.
A data packet for controlling a computer to reproduce multimedia data according to other embodiments of the present invention includes: (1) multimedia data; (2) an instruction to player software stored on the computer regarding reproduction of the multimedia data; and (3) a time code indicating when the instruction will be performed.